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Abstract: The Enacted Practices Evaluation Model (EPEM) is an accountability system for 
“promised” program practices. Evaluating programs in terms of implemented treatment (i.e. , 
enacted practices) in ways that yield valid and reliable results for accountability purposes is 
among the most vexing challenges in the field of program evaluation. Evaluations conducted 
by internal staff provide economies and efficiencies that come with organizational familiarity 
whereas external experts in the target program’s content area provide credibility and 
objectivity. This paper presents the organizational structure and operational procedures of 
an accountability system (EPEM) which adheres to the Program Evaluation Standards and 
maximizes the contributions of institutional resources and procured expert services. 



Hey! Accountability is more than outcome! Accountability is also quality of experience! 
Accountability seeks evaluation information on the extent to which a program is effective after it 
is fully implemented; but to answer that question it is first necessary to know how and the extent 
to which the program was actually implemented. Unless one knows that a program is operating 
according to design, there is no reason to expect it to provide the desired outcomes. This is not a 
question of summative versus formative evaluation, but one of quality assurance that 
instructional practices and curriculum opportunities meet certain standards of excellence. Mohr 
(1992) points out that the quality of instructional practices and opportunities enacted by a 
program is as much an accountability issue as any summative information about outcomes. 



1 Paper presented at the Sixth Annual National Evaluation Institute, Indianapolis, IN, 
July 9, 1997. 
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Evaluation of enacted program practices is a particularly acute accountability issue for 
Magnet program schools. Magnets are an “internal school choice" program which promise 
parents that their students will experience "...distinctive, high-quality course offerings not 
available in neighborhood schools" (Clewell and Joy, 1993, p. 61). The RAND Corporation 
defined school choice as a full-time educational program available to students on a voluntary 
basis, which is distinctly different from the majority of programs a school district offers. 
Beginning in 1985, PGCPS established and maintained an unusually large and diverse set of 
magnet programs as an instrument of desegregation by offering parents and students choice of 
educational opportunity. In fact, a recent court appointed panel comprehensive review of school 
desegregation in Prince George’s County found that "... the choice aspect has become dominant 
in the eyes of parents and the public, even as the school system tries hard to maintain the primacy 
of the desegregation purpose." (Peterkin, Rossell, Shoenberg, & Trent, 1997, page xiv-xv). 

The Magnet emphasis on program practices and opportunities require an accountability 
assessment system which evaluates the scope and quality of program practices and opportunities 
actually enacted. This is the purpose of the Enacted Practices Evaluation Model accountability 
assessment system developed by Research, Evaluation and Accountability that is described in 
this document. 
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Enacted Practises Evaluation Methodology: What is it? 

According to one of the earliest of modem evaluation theorists, Michael S. Scriven, 
evaluations can be designed and used in two ways to make educational programs and educational 
products more responsive to community needs: 

Evaluation may be done to provide feedback to people who are trying to improve 
something (formative evaluation); or to provide information for decision-makers 
who are wondering whether to fund, terminate, or purchase something 
(summative evaluation). (Scriven, 1980, pp. 6-7). 

More recently, Scriven (1986) and many other leaders in educational evaluation (Shadish, Cook, 
& Leviton, 1991), have elaborated the role and types of formative evaluation methodology, 
softening the distinction between formative and summative evaluation, so that some forms of 
formative evaluations can provide a summative-like accounting of the implementation success 
achieved by an educational program. The evolution in formative evaluation methodology over 
the past three decades has yielded two major types of program implementation assessment: (1) 
program monitoring or evaluation for program improvement, and, more recently, (2) measuring 
attainment of process objectives or determining achievement of means-goals. The focus of the 
former methodology is to describe the program that is occurring - whether or not this matches 
what was planned; whereas the latter formative evaluation methodology is directed at an 
assessment of implementation activities and practices featured in a program’s stated curriculum 
goals. The primary purpose of the first, more traditional formative evaluation is to provide 
assessment information for program improvement : the primary purpose of the second, 
accountability formative evaluation is to provide assessment information forjudging the success 
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of program implementation. It is the fulfillment of this latter purpose, the accountability 
assessment of the quality of program treatment (i.e., instructional practices and opportunities) 
provided by Magnet programs, that the Enacted Practices Evaluation Model (EPEM) 
methodology has been designed to deliver. Perhaps it is helpful to consider EPEM as a formal 
methodology for conducting a “certification” or "audit” evaluation of the instructional practices 
and opportunity-to-leam activities that were "promised” to parents and students by a Magnet 
program. 

In an attempt to distinguish the EPEM methodology as an accountability assessment of 
designated program practice objectives we use the term "enacted practices” rather than program 
"implemented practices.” Formative evaluation efforts traditionally focus on a program’s 
implemented practices, whether these practices are those specifically designated as the defining 
or unique characteristics of the program or not. On the other hand, "enacted" connotes doing 
what has been designated, promised, or advertised. The EPEM is an accountability system for 
designated program practices. 

The EPEM system represents the confluence of school system administration resources, 
evaluation design standards and measurement theory. Each Magnet Program evaluation study of 
enacted practices includes the following critical components: 

• Program Evaluation Standards , 

• coordination between internal and external agents, 

• identifying measurable program enacted practices, 

• methodological triangulation, and 

• dissemination and utilization procedures for evaluation results. 

- 4 - 
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Each of these components are critical to the EPEM goals for employing methodology 
which meet rigorous standards for program evaluation, take advantage of existing institutional 
resources, and yield credible and useful accounting of the quality of enacted practices for all 
types of Magnet programs. It is the opinion of the authors that, together, these components form 
an integrated structure, or logical whole, for carrying out high quality evaluation of program 
practices. 

What are, the guidelinesjor an effect ive, evaluation?. 

In 1981, a Joint Committee issued one of the most significant documents to date in the 
field of educational evaluation entitled Standardsjor Evaluation of Educational Programs , 
Projects . and Materials . This reference and its most recent update: Pro gram Evaluation 
Standards . 2 nd Edition. - consist of a set of 30 standards to be used both to guide the conduct of 
evaluation of educational programs, projects, and practices and also to judge the soundness of 
such evaluations. The 30 standards are grouped according to four attributes of an evaluation — 

A 

its utility, its feasibility, its propriety, and its accuracy. These guiding attributes played a 
powerful influence in the development of a quality assurance evaluation system to assess the 
instructional practices and curriculum opportunities provided by a variety of Magnet Programs in 
Prince George’s County Public Schools, Maryland. Guided by the “Standards "the Research, 
Evaluation and Accountability office of PGCPS developed the EPEM to provide objective and 
credible evaluation information on the quality of Magnet program implementation. 
Throughout this paper, relevant standards are highlighted in boxes like this [Z0] : 
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ZO SAMPLE STANDARD - The standard 
number, name, and description most applicable 
to each section of the text is included in a box 
like this. 



Appendix 1 list the 30 Program Evaluation Standards in summary form. These will be 
referred to simply as the 'Standards "in this document. 



Who conducts the, evaluation? 

In keeping with the goal for ensuring 
that an evaluation will serve the information 
needs of intended users, the Enacted 
Practices Evaluation Model (EPEM) uses a team approach which maximizes institutional 
resources into an external expert evaluation system [U 1 ]. 



U1. STAKEHOLDER IDENTIFICATION - 

Persons involved in or affected by the evaluation 
should be identified, so that their needs can be 
addressed. 



The Enacted Practices Evaluation Model (EPEM) leverages the procured content 
expertise of external evaluation consultants through the designated contributions of central office 
Magnet and Research staff along with school based program staff to each evaluation of program 
practices. The evaluation design, assessment implementation and measurement credibility are 
maximized through the collaboration of four parties: 

1 . Magnet School Program Office 

2. School District Research, Evaluation and Accountability (REA) office 

3. Magnet school supervisory and instructional staff from the participating school(s) 

4. External Content-Expert Evaluators 
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Each of the four (4) parties agrees to perform specific activities in the evaluation process 
[P2], The first three parties are school district employees. The external experts are contracted 

for a single evaluation effort. Figure 1 
presents an outline EPEM project schedule 
showing when and what each of the four 
EPEM team parties do in a typical Magnet 
program evaluation study. 



P2 FORMAL AGREEMENTS - Obligations of 
the formal parties to an evaluation (what is to be 
done, how, by whom, when) should be agreed to 
in writing, so that these parties are obligated to 
adhere to all conditions of the agreement or 
formally to renegotiate it. 
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Figure 1 



EPEM Project Schedule 



Stage 


Activity 


Lead Agent(s) 


1 


Select Program & Secure 
Superintendent’s Approval 


Magnet Office 


2 


Identify Measurable Practices 


REA & School Based 
Program Staff 


3 


Develop RFP to select 
external content expert 1 


REA 


4 


Review/Rate Proposals 


Proposal Review Committee 2 


5 


Proposal Recommendation(s) 
sent to School Board 


Superintendent 


6 


Award Contract and Approve 
the EPEM study 


School Board 


7 


Initiate Study, Planning 
Meetings, Orientation 


Magnet Office, REA, School 
Based Program Staff, & 
Contracted Experts 


8 


Instrumentation, Assessment 
& Analysis 


Contracted Experts and REA 


9 


Report and Presentations 3 


Contracted Experts and REA 


10 


Disaggregated Results to 
individual Magnet program 
schools in study 


REA and Magnet Program 
Supervisory Staff 



1 . Request For Proposal describes the EPEM methodology, measurable program practices which are the target of 
the particular study, and the schedule of deliverables. 



2. Proposal Review Committee has a core representation of Community Advisory Committee (CAC) members, 
REA staff, Magnet Office staff, Grant Officer, Chief Divisional Administrator from Instruction, and a school-based 
staff member from the Magnet program to be evaluated. 

3. Report and Presentations are disseminated according to the following schedule: 1) Superintendent’s Executive 
Council, 2) School Board, 3) Magnet program schools, and 4) the CAC. 
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The distribution of responsibilities are stipulated in the RFP and further clarified during 
meetings in the early stages of the evaluation process. The evaluation itself is a highly 
cooperative process and thus easily accommodates task adjustments, should it become necessary. 
Now, let’s look at each of the EPEM team agents and their primary contribution to each Magnet 
program evaluation study. 

1. Magnet School Program Office 

The Magnet School Program Office is the party responsible for initiating the evaluation 
process and using the results. This office selects which of the Magnet programs are eligible for 
evaluation, secures the Superintendent’s approval to expend evaluation resources in the target 
programs, and establishes the accountability purpose for the evaluation effort in the Request for 
Proposal (RFP). The Magnet Office closes the feedback loop by determining the use of the 
evaluation results. The bulk of this party’s efforts are at the beginning and end of the evaluation 
project. 



The Magnet Office funds the 
evaluation and thus determines the cost limit 
on external expertise. This office also 
identifies the scope and timing of the 
evaluation, in light of system needs [F2]. This 
office issues the Request for Proposal (RFP) to seek contract bids, and participates in the process 
of selecting the external expert. 



F2 POLITICAL VALIDITY - The evaluation 
should be planned and conducted with 
anticipation of the different positions of various 
interest groups, so that their cooperation may be 
obtained, and so that possible attempts by any 
of these groups to curtail evaluation operations 
or to bias or misapply the results can be averted 
or counteracted. 
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The Magnet Office is the primary agent for dissemination of program evaluation results. 



There is a wide range of interest in Magnet school evaluation results which include the 
executives and policy makers of the school system, media, courts, individual schools, 
independent researchers, and, of course, parents . While a technical evaluation report is suitable 
for some of these audiences, the Magnet 
Program Office works closely with the 
Research Office to reformat and present 
program evaluation results in a variety of 
ways to make it useful to the particular 
audience in need of information [U7]. 

Of course, one of the primary operations of the Magnet Office is to work with the 
evaluated schools subsequent to the evaluation to see that recommendations and findings are 
addressed (i.e., school/program improvement). 

2. School District Research, Evaluation and Accountability (REA) 

REA guides the evaluation design and coordinates assessment administration. REA is 
ultimately responsible for the quality of the evaluation, and is thus actively involved throughout 
the process. In a sense, REA conducts the evaluation, with the external experts serving as the 
assessment agent responsible for interpreting results in terms of content standards. It is 
important to realize that the contributions of external content expert evaluator on the evaluation 
team is not a "labor saving device ”! The EPEM requires that REA coordinate and administer all 
facets of the project operations, including close supervision and facilitation of the contracted 
external content experts, in order to deliver the information in useful forms as designated by the 




U7 EVALUATION IMPACT - Evaluations 
should be planned, conducted, and reported in 
ways that encourage follow-through by 
stakeholders, so that the likelihood that the 
evaluation will be used is increased. 



Magnet Office. 

From the inception of the evaluation process, REA is involved in the construction of the 
list of measurable enacted practices, the drafting of the RFP, and the selection of the external 
expert. The expression “Plan your work, and work your plan” is key to EPEM implementation, 
and the RFP represents the plan. A prerequisite to drafting an RFP is that REA work with the 
staff of targeted Magnet programs to clearly identify the enacted practices in measurable terms 
(more about this later). Critical components in each RFP are: an operational description of the 
Magnet program (vs. “policy" description), identification of the measurable enacted practices 
which are to be assessed by the external content experts, the study schedule, and the scope and 
nature of dissemination (i.e., reports and presentation requirements). Appendix 2 provides an 
example RFP. Once the contract is bid, REA conducts meetings with school staff and external 
experts to distribute tasks and develop the project schedule as previously listed in Figure 1. 

REA works with the external experts to develop assessment instruments. The 
instrumentation and sampling methods are designed with an adherence to those Standards which 
will promote validity, reliability, and consistency with previous evaluation efforts. Enacted 
practices evaluations identify multiple assessment sources and a variety of assessment 
procedures (e,g., surveys, focus groups, interviews, observations, and review of documents and 
extant data). Typically, evaluation data input is sought from parents, teachers and students in 
addition to the evaluators’ own observations via classroom observations, or lesson plan reviews, 
or both. 
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REA verifies that the assessment 



instruments are valid [A5]; that is, that they 
will legitimately assess the program’s 
designated practices. In addition, REA 
ensures that they are reliable [A6]; 
specifically that a sufficiently large, and 
unbiased, sample is assessed. 



A5 VALID INFORMATION - The information 
gathering procedures should be chosen or 
developed and then implemented so that they 
will assure that the interpretation arrived at is 
valid for the intended use. 



A6 RELIABLE INFORMATION - The 

information gathering procedures should be 
chosen or developed so that they will assure 
that the information provided is sufficiently 
reliable for the intended use. 



REA’s knowledge of the effectiveness of previous assessment efforts in PGCPS is 
invaluable in guiding the external experts to gather a sufficient and worthwhile pool of data. 
Often the data collection period is constrained by time demands such as standardized testing, 



thus it is particularly important to plan ahead, 
gather all information as efficiently as 
possible, and then get out of the school 
quickly [FI]. 



FI PRACTICAL PROCEDURES - The 

evaluation procedures should be practical, to 
keep disruption to a minimum while needed 
information is obtained. 



During the data collection phase, 

REA assists the external experts to facilitate 
speediness. Although the external experts 
conduct all specialized or interactive 
activities (i.e., focus groups, interviews, classroom observations), REA staff may prepare, 
administer, and/or conduct data entry of surveys, for example. All raw data is available to both 

- 12 - 



A7 SYSTEMATIC INFORMATION - The 

information collected, processed, and reported 
in an evaluation should be systematically 
reviewed and any errors found should be 
corrected. 
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parties. Although the external expert is responsible for data analysis, REA performs verification 
tests to assure quality [A7]. 

During the analysis and report writing stages, REA again performs a supporting role for 
the external experts. REA provides guidance regarding the report structure and level of detail, 
but the external expert is responsible for the 
production of the initial report. REA 
ensures that sufficient detail is provided to 
support subsequent reports to be developed 
by REA [U5]. Previous evaluation reports 
are provided by REA as a guideline. 

When the evaluation results are presented to the various stakeholders, REA advises the 
external expert in dealing with each audience. REA is present at all presentations and available 
to answer questions, etc. 

Subsequent to the external expert’s report preparation, REA generates reports on an as- 
needed basis, in various formats, depending upon the audience. These might include summative 
reports or meta-analytical reports of the magnet programs, for example. 



U5 REPORT CLARITY - Evaluation reports 
should clearly describe the program being 
evaluated, including its context, and the 
purposes, procedures, and findings of the 
evaluation, so that essential information is 
provided and easily understood. 
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3. Magnet school supervisory and instructional staff from the participating school(s) 

School program staff inform all evaluation parties of operational priorities and 
instructional context; in this, they serve as a “reality check” and provide vital information used in 
describing operational program practices and opportunity-to- learn features of the Magnet 
program. 

REA enlists the full participation of the school-based Magnet program staff through full 
disclosure of the accountability assessment nature of the EPEM study of their program [P5], 
Although this often yields some anxious 
moments, inclusion of program staff 
throughout the process and demonstration of 
how their input is used have proven time and 
again a vital component to the successful 
completion of each study undertaken. 

School staff provide general information such as annual School Improvement Plans in 
addition to the magnet-specific information [Al]. School staff also provide access to 

assessment sources. For example, class 
schedule information is necessary if 
classroom observations are to be scheduled. 

If parent surveys are to be sent home with 



Al PROGRAM DOCUMENTATION - The 

program being evaluated should be described 
and documented clearly and accurately, so that 
the program is clearly identified. 



— 

P5 COMPLETE AND FAIR ASSESSMENT - 

The evaluation should be complete and fair in its 
examination and recording of strengths and 
weaknesses of the program being evaluated, so 
that strengths can be built upon and problem 
areas addressed. 



students, school staff handles this. Access to students, teachers, parents, and lesson plans are all 



coordinated by school staff. 



4. External Content-Expert Evaluators 

The external experts provide much of the assessment content, measurement standards, 
and results interpretation. By incorporating the expertise of an outside party on a short-term 
basis, several goals can be accomplished. 

First, an increase in perceived credibility can be enjoyed due to the impartiality of the 
external party. These contracted evaluators are perceived as providing objectivity, because they 
have no stake in the positive or negative outcome of the evaluation. Evaluations conducted 
exclusively by internal parties, on the other hand, are open to criticisms of bias, rightly or 
wrongly. 

Second, by bringing in experts who are familiar with other similar programs, the 
evaluation value is increased with minimum cost [F3]. A thorough evaluation of a given magnet 
program would, for example, include comparisons to other similar academic programs. The cost 
of launching an investigation of other 
programs across the country would be 
prohibitive. A content expert, however, 
provides this knowledge inherently. The 
evaluation time and money can be spent on gathering detailed information on the program under 



F3 COST EFFECTIVENESS - The evaluation 
should be efficient and produce information of 
sufficient value, so that the resources expended 
can be justified. 



evaluation to maximum benefit. 



The external experts work closely with REA throughout the process. Their duties include 
reviewing the list of enacted practices, with an expert’s eye, to ensure appropriateness and 
thoroughness. With REA’s assistance, they develop and administer the assessment instruments 
such as surveys, focus groups, interviews, and classroom observations. The external experts also 
gather relevant literature, such as state/national standards, and review school-provided literature, 
such as lesson plans and effective school plans. In conjunction with REA staff, the external 
experts analyze the collected data and consolidate the results to reflect the enacted practices. 

This culminates in a final report, which will serve as the reference for any subsequent reports 
generated by REA. The external experts make presentations of the evaluation design and 
findings to the Superintendent’s Executive Council and the Community Advisory Council. The 
School Board has the option to request a third presentation. 

Upon completion of the report and the presentations, all materials become the property of 

REA. 



Identifying Measurable Program Features 

When REA is assigned a program to evaluate, and discusses it with sponsors or staff, it is 
generally a complicated affair. The evaluator’s aim is to break the program down neatly into the 
form of measurable enacted practices, but there often seem to be so many different kinds of 
elements to consider that doing so is no easy task. With experienced help of REA staff for 
culling practices from ongoing operations, the multifaceted nature of the typical Magnet program 
must be parsimoniously organized into measurable enacted practices for research. 



Wholey defines the evaluable program as “the portion of the program for which tho 
charge have defined plausible, measurable objectives; for which there are feasible sources of 
performance data; and for which likely uses of program performance information have been 
defined.’ By definition, the evaluable portion of Magnet Programs are the unique instructional 
experiences and opportunities designed to attract enrollment by students outside the Magnet 
school’s catchment area. Accordingly, the portion of Magnet programs in need of evaluation are 
the unique instructional practices and opportunities provided students. In practice, when it 
comes to the 17 different types of Magnet programs offered in Prince George's County Public 
Schools, identifying each program’ s measurable features or evaluable program becomes quite 

challenging. 

Established programs with standardized, measurable, well-defined characteristics enable 
straightforward evaluation. Montessori programs, for example, must adhere to the Montessori 
charter. In such well-established programs, evaluation is basically a matter of “certification"; that 
is, verifying that the program instructional practices, learning environment and opportunities 
meet the established documented standards in the Montessori charter 2 . 



Meeting the evaluation standard for identifying the evaluable features becomes more 
muddied, however, in Magnet programs offering non-traditional or relatively new educational 
initiatives experiences. In many such cases, the first evaluation challenge is to identify and 

2 Formative Evaluation Stud w The Montessori Magnet Program of Prince Georg ejs 
P.nuntv. MP Spring 1996 . Research Report No. 48-9-96, REA, PGCPS, MD. 
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define the practices and opportunities which the program has enacted in an attempt to achieve 
its stated mission. This is where two vitally important features of the EPEM come into play. 

REA expert evaluation staff guiding Magnet program staff towards measurable practice 
statements; and the use of external experts in program content areas to refine the identification of 
purported enacted practices. The former activity occurs in the Request for Proposal (RFP) 
process of the EPEM, and the latter activity takes place during the initial stages of assessment 
instrument development process by the external experts who win the EPEM contract. 

Prior to each evaluation project, the 
REA office works with the Magnet School 
Program Office and the participating schools 
to develop a few overarching practice 
statements which capture the essence of the magnet program [A3]. This step is critical. It is 
important that these be sufficiently tangible to allow measurement [Al] , yet general enough to 
permit the external expert some flexibility in determining whether the manner in which these 
practices are manifested truly meets the 

designed instructional practice as well as the PROGRAM DOCUMENTATION - the 

program being evaluated should be described 

“quality standard” for the instructional and documented clearly and accurately, so that 

the program is clearly identified. 

experience the content experts believe 
appropriate. 



A3 DESCRIBED PURPOSES AND 
PROCEDURES - The purposes and procedures 
of the evaluation should be monitored and 
described in enough detail, so that they can be 
identified and assessed. 
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For example, if a “Humanities Magnet Program” promises instructional experiences that 
broaden literary perspective and literary analysis, an inadequately stated program practice might 

read: 

“All students write monthly book reports”. 

An evaluable Humanities Magnet Program feature might read something like: 



“Program requirements ensure that students 
expand their literary repertoire through an 
ongoing reading and analytical process". 



While both statements provide measurable practices, the second statement captures the goal of 
the unique program practice rather than providing a laundry list of requirements which do not 
allow qualitative assessment. An external evaluator with content expertise in “Humanities” 
instructional programs might find, for example, that students are indeed completing monthly 
book reports, but that the instructional experience does not meet “quality standards” because the 
book selections and/or the teachers’ use of the reading materials are too limited or inappropriate 
to challenge students’ analytical skills as designated by the program. 

The identification of Magnet program evaluable features is arguably the keystone in a 
successful program evaluation. The list of identified enacted program practices are included in 
the EPEM Request For Proposal (RFP) to communicate to prospective evaluation contractors the 
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evaluable portion of the Magnet program to be evaluated. Subsequently, the enacted practice 
statements guide evaluation design and assessment instrumentation selection and/or 
development. Further, the enacted practices provide a clear structure for the final reports, so that 
the reader clearly understands the nature and scope of the evaluation. The Executive Summary, 
in particular, benefits from the succinct nature of the enacted practices. 

How do you measure ‘Success. *? 

Methodological triangulation is the use of multiple methods to study a single problem or 
program, such as interviews, observations, 
questionnaires, and documents. Denzin 
(1978) explains that the logic of 
triangulation is based on the premise that 
“...Because each method reveals different 
aspects of empirical reality, multiple methods of observations must be employed. This is termed 
triangulation.” (Denzin, 1978, p. 28) [U3] 

EPEM uses a triangulation technique which investigates the enacted practices from 
various viewpoints. It employs the subjective opinions of stakeholders in the magnet program, 
such as teachers, students, and parents; as well as the judgments of external content experts who 
have no personal concern with the evaluation outcome. The input of teachers, students and 
parents may be assessed through surveys, focus groups, or a combination of both. The external 
experts may review lesson plans, interview stakeholders, and/or observe classroom situations. 



— 

U3 INFORMATION SCOPE AND SELECTION 

- Information collected should be broadly 
selected to address pertinent questions about 
the program and be responsive to the needs 
and interests of clients and other specified 
stakeholders. 



By probing multiple sources, 
evaluators can gather substantial evidence 
for their conclusions [A9]. Overlap of 
similar questions allows a comparison of 
each cohort group’s results with the results 
from other groups. Consistent results from all sources would provide strong evidence for one 
interpretation of the assessed practice. Differences between various cohort’s responses, however, 
might reflect differing concerns, motivations or impressions. Differences may be informative in 
and of themselves; they might simply reflect a bias of one group or they might indicate a need for 
better communication between groups, for example. 



A9 ANALYSIS OF QUALITATIVE 
INFORMATION - Qualitative information in an 
evaluation should be appropriately and 
systematically analyzed so that evaluation 
questions are effectively answered. 



This triangulation process helps to 
ensure that the information gathered in the 
evaluation is reliable [A6]. REA verifies that 
a sufficient sample size is probed with each 
technique. 



— 

A6 RELIABLE INFORMATION - “The 
information gathering procedures should be 
chosen or developed and then implemented so 
that they will assure that the information 
obtained is sufficiently reliable for the intended 
use 



Process Standards From External Content Experts 

When it comes to “accountability,” credibility and acceptance are key to a successful 
evaluation study. There is no single component of the evaluation study which affects credibility 
and acceptance more than the persons conducting the evaluation [U2]. 
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Through an RFP process, the consultant 
services of external evaluators who are 
content experts in the Magnet program 
selected for evaluation to serve on the EPEM 



U2 EVALUATOR CREDIBILITY - The persons 
conducting the evaluation should be both 
trustworthy and competent to perform the 
evaluation, so that the evaluation findings 
achieve maximum credibility and acceptance. 



evaluation team. 



So vou ’ve evaluated, the_Drogram — now what? 

The external expert is responsible for producing a report describing the program, the 
evaluation process, and the evaluation findings. This is the initial, complete report from which 
information for all future reports will be drawn. Responsibility for this report’s content falls to 
the external expert, rather than other 
evaluation participants, to ensure impartiality 
[All]. REA assists in the report 
development in terms of structure and clarity. 

REA’s experience with the system stakeholders helps structure the report for maximum value. 
REA’s overall guiding principle in assessing the report — as manifest throughout the evaluation 
design and execution — is that the report is crafted in such a way as to maximize its use. 



All IMPARTIAL REPORTING - Reporting 
procedures should guard against distortion 
caused by personal feelings and biases of any 
party to the evaluation, so that evaluation 
reports fairly reflect the evaluation findings. 



This report should serve as a standalone document, containing sufficient information to 
describe the program to an audience unfamiliar with the program’s details [U5]. Thus, in 
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addition to the list of enacted practices that 
describe the unique contributions of this 
magnet program, the report should include a 
brief history of the program and a description 



— 
U5 REPORT CLARITY - Evaluation reports 
should clearly describe the program being 
evaluated, including its context, and the 
purposes, procedures, and findings of the 
evaluation, so that essential information is 
provided and easily understood. 



of the environment surrounding the program [A2]. For example, the magnet program is part of a 



system-wide desegregation effort; as such, it 
is important to describe the racial 
composition of the students in the district as 
a whole as well as the racial composition 
within the magnet program. 



A2 CONTEXT ANALYSIS - The context in 
which the program exists should be examined in 
enough detail, so that its likely influences on the 
program can be identified. 



The report also contains a description of the assessment instruments, including copies of 
surveys or focus group questions. The methodology for gathering assessment information and 
the observed actual response rate are included. The analytical/statistical techniques for 
processing the gathered data are detailed. Conclusions, both positive and negative, are clearly 
identified [P5]. The primary findings are 
described in terms of the enacted practices 
identified at the outset of the evaluation 
effort. Incidental findings are also included. 



— 

P5 COMPLETE AND FAIR ASSESSMENT - 

The evaluation should be complete and fair in its 
examination and recording of strengths and 
weaknesses of the program being evaluated, so 
that strengths can be built upon and problem 
areas addressed. 



The specific information sources 
(e.g., survey items, focus group comments) 
for each conclusion and finding are specified 
[A10]. (See "How do you measure success" section for details.) 



A10 JUSTIFIED CONCLUSIONS - The 

conclusions reached in an evaluation should be 
explicitly justified, so that stakeholders can 
assess them. 



After both parties (REA and external experts) are satisfied with the initial report, the 
external expert prepares a presentation of the 
evaluation results [P6]. REA provides 
suggestions and guidelines in a rehearsal 
session some days before the first official 
presentation. 



P6 DISCLOSURE OF FINDINGS - The formal 
parties to an evaluation should ensure that the 
full set of evaluation findings along with pertinent 
limitations are made accessible to the persons 
affected by the evaluation, and any others with 
expressed legal rights to receive the results. 



The initial presentation is made to the Superintendent’s Executive Council. The report is 
distributed to Council members a week before the meeting. At the meeting, the external experts 
briefly review the evaluation findings, and a discussion period ensues. REA, Magnet Program 
Office staff, and members of the schools’ magnet staff (if appropriate) attend to field questions. 
The Council’s comments and suggestions are used to refine and clarify the report prior to further 
dissemination. REA and the external experts modify the report, if necessary. 



The refined report then follows two dissemination tracks. First, the accountability goals 
of the evaluation are satisfied by distributing the final report to the Superintendent’s Executive 
Council, the School Board, and the Community Advisory Council (CAC) for the Magnet School 
Program. The School Board may or may not request a presentation of the results in addition to 
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the written report. The CAC always receives a presentation. This dissemination track permits a 
top-down (i.e., accountability) approach to dealing with study findings; that is, these supervisory 
bodies may demand programmatic changes or provide funding to enable such changes. 



Two recent examples of top-down changes emanating from such an evaluation are 
illustrative. First, an evaluation of the Science, Mathematics and Technology magnet program 
revealed shortcomings in the available computer hardware which constrained the integration of 
technology into the teaching of math and science. Subsequently, all the schools in this program 
received upgraded hardware. 

A second example comes from our Montessori evaluation. One finding was that parent 
participation was inconsistent between sites. As a result, a teacher network among the 5 
buildings was established to share best practices for parent participation. 

The second dissemination track has a feedback loop approach. In this case, REA staff 
and Magnet Program staff jointly provide detailed disaggregated feedback to the Magnet staff 
and teaching staff at the participating school(s). This provides an avenue for school-initiated 
changes. Routinely, the staff incorporates EPEM study results to modify, where appropriate, 
their annual “School Improvement Plan”. 

An example of the effectiveness of the feedback loop comes from our Montessori 
evaluation. Evaluators determined that longer uninterrupted work periods would enhance the 
effectiveness of the program. Since the evaluation, all school-based teams have been working 



toward some form of block scheduling to alleviate some of the interruptions in the work period. 



This completes the regular dissemination of EPEM results. The information then takes 
on a life of its own, appearing in various guises. REA may select specific aspects of the 
evaluation for inclusion in other reports, such as combining the EPEM results with student 
achievement results which are dissaggregated by Magnet program (i.e., summative). In the end, 
Lawrence B. Mohr (1992) offers very good advise on why evaluations should strive for quality in 
regards to dissemination and use of results: 

“Program evaluations cannot be expected to determine the outcomes of policy 
processes. These generally involve politics, bargaining, and trade-offs with other 
policies, .... The function of and evaluation is only to provide good information. 
However, that is a supremely important function. It is true that data will not always 
carry as much weight as pure rationality would dictate. What is important to 
recognize is that a sound evaluation with implications in favor of a certain policy 
alternative will provide powerful ammunition to the friends of that alternative and 
severe problems for its foes. That is why sound evaluation is important, where sound 
evaluation means recognizing the full scope of the task and doing the whole job 
well.”(Mohr, 1992, page 209). 



Summary 

The authors are pleased with the Enacted Practices Evaluation Model as an accountability 
assessment system for promised instructional practices and opportunity-to-leam activities. We feel 
that we have successfully instantiated most of the Standards. Our evaluation projects clearly benefit 
from the credibility afforded by the participation of external experts. The team approach keeps the 
out-of-pocket cost of external expertise down. Not only does REA’s continual involvement leverage 
the procured contributions of external expertise, but it also ensures quality control at each step of the 
evaluation process. The careful definition of measurable enacted practices helps to focus the entire 
evaluation effort, and also provides a clear framework with which to communicate the evaluation 
results to all stakeholders. Development efforts in the future will focus on improving methods to 
identify programs’ enacted practices for evaluation and standardizing our reporting formats. 
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Appendix 1 

The Program Evaluation Standards 



THE PROGRAM EVALUATION STANDARDS 

Sound evaluations of educational programs, projects, and materials in a variety of settings should 
have four basic attributes: 

• Utility • Propriety 

• Feasibility • Accuracy 

The Program Evaluation Standards , established by sixteen professional education associations, 
identify evaluation principles that when addressed should result in improved program evaluations 
containing the above four attributes. 

Dr. James R. Sanders, Chair 
The Joint Committee on Standards 
for Educational Evaluation 
The Evaluation Center 
Western Michigan University 
Kalamazoo, Michigan 49008-5178 
616-387-5895 



This tear out is not copyrighted material. Reproduction and dissemination are encouraged. 



Utility 



The utility standards are intended to ensure that an evaluation will serve the information needs of 

intended users. 

U1 Stakeholder Identification Persons involved in or affected by the evaluation should be 
identified, so that their needs can be addressed. 

U2 Evaluator Credibility The persons conducting the evaluation should be both trustworthy and 
competent to perform the evaluation, so that the evaluation findings achieve maximum credibility 
and acceptance. 

U3 Information Scope and Selection Information collected should be broadly selected to address 
pertinent questions about the program and be responsive to the needs and interests of clients and 
other specified stakeholders. 

U4 Values Identification The perspectives, procedures, and rationale used to interpret the findings 
should be carefully described, so that the bases for value judgments are clear. 

U5 Report Clarity Evaluation reports should clearly describe the program being evaluated, including 
its context, and the purposes, procedures, and findings of the evaluation, so that essential 
information is provided and easily understood. 

U6 Report Timeliness and Dissemination Significant interim findings and evaluation reports 
should be disseminated to intended users, so that they can be used in a timely fashion. 

U7 Evaluation Impact Evaluations should be planned, conducted, and reported in ways that 
encourage follow-through by stakeholders, so that the likelihood that the evaluation will be used 
is increased. 




Sage Publications, Inc. 

2455 Teller Road 
Thousand Oaks, CA 91320 
805-499-0721 




JCSEE PR-1994 
Approved by the American 
National Standards Institute 
as an American National 
Standard. Approval date: 

March 15, 1994@ 
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Feasibility 



The feasibility standards are intended to ensure that an evaluation will be realistic, prudent, 

diplomatic, and frugal. 

FI Practical Procedures The evaluation procedures should be practical, to keep disruption to a 
minimum while needed information is obtained. 

F2 Political Viability The evaluation should be planned and conducted with anticipation of the 
different positions of various interest groups, so that their cooperation may be obtained, and so 
that possible attempts by any of these groups to curtail evaluation operations or to bias or 
misapply the results can be averted or counteracted. 

F3 Cost Effectiveness The evaluation should be efficient and produce information of sufficient 
value, so that the resources expended can be justified. 



Propriety 



The propriety standards are intended to ensure that an evaluation will be conducted legally, 
ethically, and with due regard for the welfare of those involved in the evaluation, as well as those 
affected by its results. 

PI Service Orientation Evaluations should be designed to assist organizations to address and 
effectively serve the needs of the full range of targeted participants. 

P2 Formal Agreements Obligations of the formal parties to an evaluation (what is to be done, 
how, by whom, when) should be agreed to in writing, so that these parties are obligated to 
adhere to all conditions of the agreement or formally to renegotiate it. 

P 3 Rights of Human Subjects Evaluations should be designed and conducted to respect and 
protect the rights and welfare of human subjects. 

P4 Human Interactions Evaluators should respect human dignity and worth in their interac- 
tions with other persons associated with an evaluation, so that participants are not threatened 
or harmed. 

P5 Complete and Fair Assessment The evaluation should be complete and fair in its exami- 
nation and recording of strengths and weaknesses of the program being evaluated, so that 
strengths can be built upon and problem areas addressed. 

P6 Disclosure of Findings The formal parties to an evaluation should ensure that the full set of 
evaluation findings along with pertinent limitations are made accessible to the persons 
affected by the evaluation, and any others with expressed legal rights to receive the results. 
P7 Conflict of Interest Conflict of interest should be dealt with openly and honestly, so that it 
does not compromise the evaluation processes and results. 

P8 Fiscal Responsibility The evaluator's allocation and expenditure of resources should reflect 
sound accountability procedures and otherwise be prudent and ethically responsible, so that 
expenditures are accounted for and appropriate. 









Accuracy 



The accuracy standards are intended to ensure that an evaluation will reveal and convey technically 
adequate information about the features that determine worth of merit of the program being 
evaluated. 

A1 Program Documentation The program being evaluated should be described and docu- 
mented clearly and accurately, so that the program is clearly identified. 

A2 Context Analysis The context in which the program exists should be examined in enough 
detail, so that its likely influences on the program can be identified. 

A3 Described Purposes and Procedures The purposes and procedures of the evaluation should 
be monitored and described in enough detail, so that they can be identified and assessed. 

A4 Defensible Information Sources The sources of information used in a program evaluation 
should be described in enough detail, so that the adequacy of the information can be assessed. 
A5 Valid Information The information gathering procedures should be chosen or developed and 
then implemented so that they will assure that the interpretation arrived at is valid for the 
intended use. 

A6 Reliable Information The information gathering procedures should be chosen or developed 
and then implemented so that they will assure that the information obtained is sufficiently 
reliable for the intended use. 

A7 Systematic Information The information collected, processed, and reported in an evaluation 
should be systematically reviewed and any errors found should be corrected. 

AS Analysis of Quantitative Information Quantitative information in an evaluation should be 
appropriately and systematically analyzed so that evaluation questions are effectively answered. 
A9 Analysis of Qualitative Information Qualitative information in an evaluation should be 
appropriately and systematically analyzed so that evaluation questions are effectively answered. 
AIO J ustifled Conclusions The conclusions reached in an evaluation should be explicitly justified, 
so that stakeholders can assess them. 

All Impartial Reporting Reporting procedures should guard against distortion caused by 
personal feelings and biases of any party to the evaluation, so that evaluation reports fairly 
reflect the evaluation findings. 

A12 Metaevaluation The evaluation itself should be formatively and summatively evaluated 
against these and other pertinent standards, so that its conduct is appropriately guided and, on 
completion, stakeholders can closely examine its strengths and weaknesses. 

Guidelines and illustrative cases to assist evaluation participants in meeting each of these standards 
are provided in The Program Evaluation Standards (Sage, 1994), The illustrative cases are based in 
a variety of educational settings that include schools, universities, medical and health care fields, the 
military, business and industry, the government, and law. 
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Appendix 2 



Example Evaluation Study RFP 
K-8 Creative and Performing Arts Magnet Program 

Request for Proposal 

The Board of Education of Prince George’s County 
Research, Evaluation and Accountability 
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The Board of Education of Prince George's County 

Research, Evaluation and Accountability 



Fnrmative Evaluation Study 
K-8 Creative and Performing Arts Magnet Program 

Request for Proposal 
September 1996 



Introduction The Research, Evaluation and Accountability (REA) office of The Board of 
Education of Prince George’s County, in conjunction with the Magnet School Office 
is conducting a formative evaluation of the K-8 Creative and Performing Arts 
Magnet program. This effort involves the on-site qualitative evaluation by the 
awardee of the contract. 

Purpose Prince George's County Public Schools (PGCPS) is seeking proposals from 
experts/institutions to complete a qualitative/formative evaluation of the educational 
program within the School District’s K-8 Creative and Performing Arts Magnet 
School.(Thomas G. Pullen) 



Program 

Description The Creative and Performing Arts at Thomas Pullen is open to students in 
Kindergarten through eight grade. It is designed to develop the interests and talents 
of students in the arts and features an enhanced interdisciplinary academic program 
that encourages creative and artistic expression. Experience and training are 
designed to challenge and develop skills of all students, as well as to provide 
exceptional opportunity for artistically talented students. 

The curriculum provides in-depth experiences in each art discipline, plus related 
arts experiences and an infusion of the arts in the overall curriculum. The arts are 
provided as an integral part of a strong academic program. 

The K-8 Creative Arts School follows the general curriculum guidelines that are used 
in all Prince George’s County public elementary and middle schools. Basic 
instruction is provided in reading, mathematics, English, science, and social 
studies, as well as specialized instruction in the arts - art, drama, music, dance, 
physical education, creative writing, media production, literary arts, and related 
computer lab experiences. 



Objective 


The principal investigator(s), using their expertise in Art Education, will assess the 
level of program implementation enacted within the participating school and 
classrooms. Specifically, the qualitative evaluation will assess the degree to which 
the Prince George’s County Public School’s curricula are put into actual classroom 
practice Assessment data will be gathered through on-site structured observations 
interviews, surveys, etc. Final analysis will show the degree to which declared 
practices have been enacted, as well as a summary of “best practices observed. 

The scope of this evaluation is an examination of how well the Thomas G. Pullen 
School has implemented the creative and performing arts magnet program. The 
program curriculum and instructional practices specifically associated with the 

Thomas Pullen program are: 

1. The integration of the creative and performing arts with academic 
learning, specifically: 

A. Academic instruction enhances the creative and performing arts; 

B. Creative and performing arts instruction enhances academic 

learning. 

2. The multicultural thread is woven through both the academic and 
creative and performing arts program. 

3. Students develop a knowledge of and appreciation for the arts. 

Therefore, the evaluation focus is to measure Pullen's ability to integrate the 
creative and performing arts with academic learning, develop an interdisciplinary 
approach to instruction stressing academic achievement, creativity, artistic 
expression, and an appreciation for cultural diversity. 


Consultant 

Services 

Required 


On behalf of the REA and the Magnet Office, the principal investigator shall 
perform the following services in the formative evaluation of the K-8 Creative and 
Performing Arts Magnet Program: 

1 Demonstrate a high level of understanding of the Creative and Performing 

Arts Magnet Program, at both the elementary and middle school levels, its 
design, objectives, enrollment procedures, curriculum, teaching strategies 
staffing and staff development. (This may require school visits, office 
visits, interviews, observations, etc.) 



2. Consult with program coordinator(s) and REA to verify findings from #1 . 

3. Present to REA copies of all data collection instruments (i.e. surveys, 
observation forms, interview questions, etc.) for approval to implement. 

4. Design an evaluation time table and present to REA a detailed schedule. 

5. Design and submit to REA prototypes of all reporting formats of graphs, 
tables and charts for approval to implement. 

6. Prepare in advance of data collection activities, a prototype report outline, 
introduction section and table of contents of the summary report for 
approval to REA. 

7. Conduct a structured field study data collection at Thomas G. Pullen. The 
structured field study may be completed using REA approved surveys, 
focus groups, observations, interviews, video tapings, etc. 

8. Collect and compile measurable data that yields a quantifiable 
representation of the enacted practices pertaining to program, teaching 
strategies, and curriculum components. 

9. Analyze data for the degree of agreement between declared practices and 
those practices actually observed. Report on the degree of agreement 
between declared and observed practices. Using the declared practices as 
the goal, list the strengths and weaknesses of the program. 

1 0. Deliver a first draft, a revised second draft, and a final report document. 
All documents shall be delivered in both hard-copy and soft-copy (Word 
Perfect 6.1) format. 

1 1 . Prepare and give a presentation with overheads (or computer presentation 
graphics) to die School System’s Executive Council, the Board of 
Education, and the Community Advisory Council. 

Schedule The final report document shall be delivered to REA no later than close of 

business on May 15, 1997. 



O 
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The Board of Education of Prince George's County 

Research, Evaluation and Accountability 

Formative Evaluation Study 
K-8 Creative and Performing Arts Magnet Program 

Request for Proposal 

APPLICATION CHECKLIST 



I. Cover Sheet 

II. Abstract (100 words, maximum) 

Narrative (5 pages, maximum) 

Expertise and background of institution and/or investigation 
Evaluation questions to be addressed 
Data collection procedures and methodology 
Instrumentation 

Schedule and Time line of Activities 

IV. Budget 

V . Reference Letters (3 , minimum) 

VI. Resumes (of all assigned staff and indicating Principal Investigator(s)) 

VII. Assurances and Certifications 
EOE 

Drug-free Workplace 
Disclosure of Lobbying Activities 
Debarment, Suspension, Ineligibility 

VIII. Submit completed proposal to REA by 5:00 p.m., Friday, November 15, 1996 



0 



The Board of Education of Prince George's County 

Research, Evaluation and Accountability 

Formative Evaluation Study 
K-8 Creative and Performing Arts Magnet Program 

Request for Proposal 

COVER SHEET 



Organization Name: 



Organization Address: 



Organization Telephone Number: 



Name(s) of Principal Investigator(s) 



Amount of Request: $ 



Signature of Authorized Representative : 
Title : 

Date Signed : 
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